DESN2003

Research for Innovation

Hongshan Guo

Week 7 Part 1: Data Collection Methods

Learning Objectives

By the end of this session, you will be able to:

  1. Distinguish primary vs. secondary data sources and when to use each
  2. Design effective questionnaires and survey instruments
  3. Select appropriate data collection methods for your research question
  4. Apply observation techniques to a real-world scenario

Artifact Output: What You Leave With

Today’s Deliverable

By end of class, you’ll have completed the Observation Design Exercise (jay-walking study):

  • Observation method selection with justification
  • Behavior categories defined
  • Reliability strategy outlined

This directly prepares you for Doc 0.2 (Methodology).

Introducing Data before its Collection

Types of Research Data Collection (Source)

  • Primary Data Source
    • 1st hand information
    • not changed by any individual
    • not published yet
    • directly collected by authors
  • Secondary Source
    • published data
    • what literature review is based on
    • reviewed by authors(you)

Examples of data collection methods by category

  • Primary Data Collection Examples
    • Questionnaires
    • Interviews
    • Focus Group Interviews
    • Observation
      • participant/non-participant
      • aware/non-aware
    • Survey
    • Statistical Methods
    • Experimental Methods
  • Secondary Data Collection Examples
    • Published Papers/Sources
    • Databases
    • Books
    • General websites
    • Unpublished personal records
    • Census data/population statistics

Primary Data Collection Methods: Designing Questionnaires

  • a set of questions and secure answers from respondents
  • often analyzed by statistical methods
  • consistency in questionnaires make cross-sectional analysis easy

Types of questions designed to measure variables in survey:

  • Close-end questions
    1. Two-option aka dichotomous scales
    2. More than two options: Nominal-polychromous
    3. Ordinal-Polytomous scale
    4. Continuous or bounded types
  • Open-end questions
    • sentence completion
    • open-ended questions with free text responses

Polytomous Variables, aka Options for Multiple Choice Questions

Statistical term that refers to a categorical variable with more than two possible categories or levels

Common in: Survey research ,healthcare, market research and education

Characteristics:

  • Categorical data: limited/distinct values or categories that are mutually exclusive
  • Always more than two options(binary)
  • Ordered and unordered (natural hierarchy)
  • Statistical analyses: e.g. logistic regression, cluster analysis for patterns and trends
  • Convertable to binary variables

Purpose:

  • Understanding complex phenomena: patterns and trends in the data collected
  • Provide insights to better categorization of data
  • Statistical analyses: Relationships between variables: e.g. logistic regression to predict probability
  • Market segmentation: e.g. customers to preference, behavior, demographic groups

Polytomous Variable (Continued)

When to use Polytomous Variables

  • Measuring attitudes or perceptions: for more nuanced perceptions
  • Putting definitions on categorizzing data: greater variability
  • Analyzing relationships between variables: job satisfication & job performance
  • Need to capture complex phenomena

Limitations

  • Power of explanation limited by available options
    • e.g. satisfactory survey & number of enrolled students, what about change?
  • subjectivity from questionnaire design and interpretation of results
  • response bias
  • small sample sizes
    • calculating the appropriate size of population needed
    • amount of respondents needed to reach statistical significance (reject null hypothesis, not covered by current class, leverage sample size calculator online)

Primary Data Collection: Designing Questionnaires

Face-to-face, paper-and-pencil or remote, make sure your data collected is: - well-organized and - easily accessible for analysis.

General rules for constructing a questionnaire:

Dos:

  • questions should be short and simple
  • provide clear navigation to avoid difficulty in reading and motivate answering
  • use positive sentences
  • add open-answer possibility after provideing listed answers
  • improve reliability by selecting appropriate words
  • explain importance of the questionnaire
  • order your questions to solicit the right answers (sensitive to follow concrete/innocuous ones)

Do-nots:

  • use more than one question (double-barreled) in one item
  • make assumptions for the respondents
  • lead the respondent to answers with clues, suggestions and hints

Steps involved in designing a questionnaire

Primary Data Collection: Interviews

Face-to-face and remote (telephone/zoom) interviews and merit/demerits (Kabir, 2016).

Good for complex or sensitive concepts and need detailed and high-status information (Frechtling, 2020).

Types of interviews by structure:

  • structured interviews: standardized questions that are pre-prepared
  • semi-structured interviews: conducted based on guide but goes beyond list of questions
  • unstructured interviews: informal, casual conversations

Rundown of an interview process

Primary Data Collection: Focus Group Discussion (FGD)

  • Mixture of interview and observation
  • Used to discover human behavior, attitudes and respondents facing a particular concept
  • 6-12 people in each group with shared characteristics
  • Mediator aims to stimulate and discover the behavior of the participats and reasons for each behavior using the social dynamic of the group

FGD: Strengths and Weaknesses

Strengths:

  • discover social, health and cultural concepts
  • literacy of individuals non-issue
  • suitable to explore complex subjects
  • useful to develop hypotheses

Weaknesses:

  • expensive and time-consuming
  • privacy risks
  • confined by readiness of facilitator/mediator
  • domination of limited individuals in focus group (Frechtling, 2002, Kabir, 2016)

Survey and questionnaire: What’s the difference?

  • Questionnaire is the written set of questions.
  • Survey is both the set of questions and the process of collecting, aggregating and analyzing the responses from those questions.

Survey: Example survey accompanying sheet

Good & Bad Survey Questions: Let’s try out how to conduct surveys

Context: Surveying respondents on their religious beliefs and life styles

  1. How religious are you?
  • vague, not sure what is being asked
  • How would you rate your level of spirituality? Pick a value between 0 to 4
  1. What do you think about smoking on campus?
  • vague, too many possible answers
  • Do you believe that all buildings on campus should be designated as smoke-free?
  1. How important is spirituality in your life? Pick a value between 1(not at all) to 5 (Very much so), (3: somewhat important).
  • leading, suggests that spirituality is important
  • What role does spirituality play in your life? Pick between 1(not important) to 5(very important) with 3 being somewhat important

Recap: Tips for effective surveys:

  1. Avoid ambiguity
  2. Avoid leading questions
  3. Avoid lengthy surveys and very long responses
  4. How will your initial questions influence answers to subsequent ones?
  5. Think carefully about sampling techniques
  6. Seek to achieve highest rate possible
  7. Standardize administration procedures
  8. Guarantee anonymity (or confidentiality at minimum)
  9. Seek measures of reliability
  10. Assess validity.

Primary Data Collection: Case Studies

  • Opportunity to investigate issues deeply and descriptively.
  • technically not a research method
  • combination of various methods to form proper understanding of the proposed case:
    • gather data through qualitative methods including interviews and surveys
    • acquire secondary-sourced data e.g. essays and diaries for analysis
    • personally provided notes can be also utilized alongside official ones

Information Sources for Case Studies

  • Direct observation from subjective evaluations
    • Single or group of observers
  • Participant observation
    • Researcher participate in the setting like other under-study people and observe happenings from a closer perspective
  • Conduct interviews with survey-type questions that are structured based on more open-ended question sets.
  • Use various census and survey records, newspapers, letters, instruments, tools, etc.

Merit and Challenges of working with Case Studies

Pros/Merits:

  • Combines the strengths of multiple research methods
  • Consider research from various time frame: past, present and future
  • Provide explanation about the changes and impacting factors that are not readily available

Cons/Challenges:

  • Complex processes, time consuming and expensive
  • No clear limit on when to stop collecting data
  • The assumption taken may not always be realistic or data tested in that context
  • Usually requires expert and trained conducting teams
  • Over-interpretation and over-generalizing issues can happen (Taherdoost, 2021)

Primary Data Collection: Experimental

  1. Laboratory/Controlled:
    • Highest control over study design and process,
    • gain precise and accurate data
  2. Field:
    • real-life situation,
    • variables are manipulated still but
    • your control is lower than Scenario 1
  3. Natural experiment:
    • no control over variables/environmental setting
    • very low reproducibility

Secondary data collection methods

Data gathered from published sources.

Challenges of Data Collection Process

  1. Location of data collection

    • neutral location
    • participants to feel free to provide their responses
  2. Literacy of Participants and Language of Questions

    • Design of questions is appropriate for the literacy level of participants
    • may require pilot tests to confirm (added costs)
  3. Timing

    • Duration of test needs to be long enough to yield reasonable results
    • Short enough to maintain the engagement of the participants
  4. Sensitivity of Data

    • Privacy of the Participants (Unique identifier)
    • Protecting personal information through promises and icebreakers and examples

Activity: Observation Design Exercise

Ethics in Observation Studies

Before You Observe: Ethics Checklist

  1. Public vs. Private Space — Observing in truly public spaces (streets, plazas) generally doesn’t require consent. Private spaces always do.

  2. Reasonable Expectation of Privacy — Even in public, some behaviors have privacy expectations (e.g., conversations, personal moments).

  3. Recording — Photos/video raise the bar. In HK, recording in public is legal but publishing identifiable images may require consent.

  4. Vulnerable Populations — Extra care with children, elderly, people in distress.

  5. No Intervention — Observation means watching, not influencing. Don’t create situations to observe.

  6. Data Protection — If you record identifiable data, you have obligations under PDPO (Personal Data Privacy Ordinance).

When in doubt: Describe your observation plan to your instructor before starting.

Architecture/Urban Example: Observing Public Space Use

Research question: How do people use seating in Hong Kong’s privately-owned public spaces (POPS)?

Design Element Observation Approach
Location 3 POPS in Central (Exchange Square, Cheung Kong Center, IFC)
Behavior categories Sitting, standing, eating, phone use, socializing, transiting
Time sampling 15-min observation blocks, 3x daily (morning, lunch, evening)
Recording Tally sheet + behavioral mapping on floor plan
Ethics Public space, no recording of faces, aggregate data only

This type of study informs urban design guidelines and policy.

Let’s Practice: Designing an Observation Study

Several years ago, a group of students at University of Central Arkansas conducted a study in which they observed the rate at which cars failed to stop at a campus stop sign and recorded whether the car had a student parking decal or a faculty/staff parking decal. This is obviously not fitting for Hong Kong context. Let’s perhaps picture a study of the rate of jay-walking at a traffic light instead - and record whether the pedestrian who crossed is a student/staff/tourist/local resident. Use what we have covered today to answer questions 1-7:

  1. Which method of observation would be best? Justify your answer. Hint: back to participant/direct observations.
  2. How would you schedule observations?
  3. Define the categories of behavior that you would observe
  4. Describe how you would optimize and measure the reliability of observations, including the use of independent observers and calculation of interobserver agreement.
  5. Describe how you could use equipment for observation rather than human observers, what are the advantages and disadvantages?
  6. Describe how you might use public records to answer the same research question. What might be some limitations of this approach
  7. Describe how you might use a survey method to answer the same research question. What might be some limitations of this approach?

References

  1. Frechtling, J. (2002). An overview of quantitative and qualitative data collection methods The 2002 user-friendly handbook for project evaluation (pp. 43-62).
  2. Hox, J. J., & Boeije, H. R. (2005). Data collection, primary versus secondary Encyclopedia of social Measurement (Vol. 1): Elsevier.
  3. Data collection challenges (2005).
  4. Kabir, S. M. S. (2016). Methods Of Data Collection Basic Guidelines for Research: An Introductory Approach for All Disciplines (first ed., pp. 201-275).
  5. Olsen, W. (2012). Data collecti on: Key debates and methods in social research (Vol. 1): Sage.
  6. Pandey, P., & Pandey, M. M. (2015). Research Methodology: Tools and Techniques (Vol. 1). Romania: Bridge Center.
  7. Rimando, M., Brace, A. M., Namageyo-Funa, A., Parr, T. L., Sealy, D.-A., Davis, T. L., . . . Christiana, R.W. (2015). Data collection challenges and recommendations for early career researchers. The Qualitative Report, 20 (12), 2025-2036.
  8. Taherdoost, H. (2016a). How to design and create an effective survey/questionnaire; A step by step guide. International Journal of Academic Research in Management (IJARM), 5 (4), 37-41.
  9. Taherdoost, H. (2016b). Measurement and scaling techniques in research methodology; survey/questionnaire development. International Journal of Academic Research in Management (IJARM),6 (1), 1-5.
  10. Taherdoost, H. (2016c). Sampling methods in research methodology; how to choose a sampling technique for research. International Journal of Academic Research in Management (IJARM), 5 (2), 18-27.
  11. Taherdoost, H. (2019). What is the best response scale for survey and questionnaire design; review of different lengths of rating scale/attitude scale/Likert scale. International Journal of Academic Research in Management (IJARM), 8 (1), 1-10.
  12. Taherdoost, H. (2021). Handbook on Research Skills: The Essential Step-By-Step; Guide on How to Do a Research Project (Kindle ed.): Amazon.

Learning Objectives

By the end of this session, you will be able to:

  1. Prepare for informational interviews through professional research
  2. Craft effective outreach messages for professional networking
  3. Apply thematic analysis to qualitative data (codes → themes)
  4. Articulate the “so what?” of your findings

Reflecting on Literature Review Submissions

Common Patterns in Doc 0.1 Submissions

What typically works well:

  • Clear articulation of research focus
  • Adequate number of sources

Common areas for improvement:

  • Formatting doesn’t match discipline conventions
  • Synthesis missing — reads as summary, not analysis
  • Theoretical framework underdeveloped
  • Conclusions included (lit reviews don’t conclude — they set up your contribution)

Quick flash back on Document 1 submission

  • Research-paper like
  • ~80% of work needed to draft a research paper as it contains:
    • Abstract
    • Literature Review
    • Methodology
    • Hypothesis/Anticipated Conclusions
    • Relevant References
  • How is this different from a research proposal?

Informational Interview/Coffee Chat

Informational Interview/Coffee Chat

  • What they are:
    • Professional/Casual Networking tools
      • Potential mentors, collaborators, future job opportunities
    • Sitting down with people who have started a career
    • Gather insights and build professional relationships
  • Learning Opportunity
    • Someone’s firsthand experience
    • challenges & successes
    • career paths
    • realistic picture of what to expect in a particular field or role (Not an ask, but preferrably something that we can look at)

Setting Expectation

  • Neither should bear agenda to secure job offer
  • Learn and build connections
  • Experience informational interview as a long-term investment
  • Start building/expanding professional network
  • The group project is a teaser of what you will likely experience going further into your career (regardless of discipline)

Quick Q&A on Information interviews

  • Do you think you have a better idea of what these are by now?
  • Any questions that you may have before we dive deeper into how to design/prepare for one?

Identify Person(s) of Intersts

  • Who to reach out to?
    • Industry Leaders and Role Models
    • Alumni Network
    • Professional Associations
  • Utilizing LinkedIn:
    • Advanced Sesarch Features
      • Filter professionals by industry, company, role and even location
    • LinkedIn Groups: Join groups of related interests.
  • How to narrow scope of search?
    • Finding Common Grounds
      • Shared Interests
      • Recent Publications or Talks
    • Leverage Existing Connections: Academic/Industry
  • Demo

Crafting Effective Cold Messages

Key Components:

  1. Subject Line: Clear and engaging subject line that conveys the purpose of the message.
  2. Personalized Greetings: Use the recipient’s name and personalized remark/compliment on recent accomplishments;
  3. Introduction and Purpose: brief but effective.
  4. Purpose Statement: seek advice, insights, experiences related to field of interest;
  5. Make Connection: highlight commonalities shared with recipient
  6. Express Genuine Interest
  7. Specific Request: Call to Action with Flexibility
  8. Politeness and Conciseness
  9. Closing and Signature

Let’s checkout some examples. Give a probability that you will respond to a message that you receive on LinkedIn. (3 Examples)

Reaching Out:

Choosing the Right Platform:

  • Platform Etiquette:
    • Norms and best practices for different platforms (LinkedIn, email, professional forums)
    • Initial outreach varies accordingly
  • Professional Channels:
    • using professional channels for communication is important
    • make sure your profiles on these platforms are up-to-date and professional.
      • What do we mean by professional?

Timing and Frequency:

  • Optimal Timing:
    • best times to send messages: avoiding weekends, late nights, and major holidays
    • the likelihood of a timely response improves accordingly.
  • Follow-up Strategy:
    • follow-up strategy involves a waiting period (typically 1-2 weeks)
    • a polite follow-up message if there’s no initial response.

Reaching Out (Continued)

Managing Outreach Volume:

  • Tracking System:
    • set up a simple tracking system (like a spreadsheet) to monitor whom you’ve contacted,
    • when, and any responses or follow-up actions needed.
  • Balanced Approach:
    • ideally no mass messaging.
    • a balanced, targeted approach to outreach,
    • focusing on quality and personalization over quantity.

Dealing with Responses:

Positive Responses:

  • Immediate Acknowledgment:
    • respond promptly to positive replies,
    • expressing gratitude for the willingness to engage and
    • proposing potential times for the chat or meeting.
  • Preparation:
    • being well-prepared for the interaction,
    • with specific questions and goals in mind.

Negative or No Responses:

  • Handling Rejection:
    • not everyone will have the time or interest to respond
    • it’s not a reflection of their worth or the quality of their outreach.
    • aka it might be very frustrating.
  • Learning from Silence:
    • how would a lack of response offer learning opportunities?
    • should you change your outreach strategy or message content?

Know Your Client (KYC)

Do your homework so that you do not make mistakes.

Research Basics

  • Background Check: reviewing LinkedIn profile, professional biographies, recent publications, news mentions.
    • Thorough professional research before outreach
  • Understanding their work:
    • familiarize with the professional’s role/projects/contributions
    • understanding the context and significance of their work

Contextual Understanding

  • Know the Industry: major trends, challenges and opportunities
  • Company Culture: culture and values of where they work/associate with for talking points

Know Your Client (KYC) Cont’d

Preparing Questions: Structured/Semi-structured

  • Informed Questions: Informed and Open-Ended, reflecting their research and curiosity about the professional
  • Personalized Inquiry: avoid overly general or easily Google-able questions
  • Object-Oriented: what do you want to achieve through the interviews
    • get first-hand critique on the discipline
    • know what day-to-day is like at the particular industry
  • Set an agenda: what will happen if you don’t have one?

Activity: Research a Professional (10 min)

  • Do background research on a potential interviewee you might be interested in reaching out to
  • you should cover at least the following:
    • their professional role
    • list out companies they worked for
    • their career highlight/work most proud of
    • identify interviews/podcasts/presentation where they gave public opinion and based on those
    • come up with two questions that you think is relevant to ask

Meet Your Client

Showing Engagement:

  • Active Engagement:
    • active listening,
    • showing interest through body language (in person or on video calls), and
    • asking follow-up questions based on the discussion.
  • Note-taking:
    • taking brief notes during the conversation (with permission) to capture key points and advice,
    • demonstrates engagement and respect for the professional’s insights.

Respect and Professionalism:

  • Time Management:
    • be mindful of the agreed-upon time for the conversation and
    • to avoid overextending unless the professional indicates they’re willing to continue.
  • Confidentiality:
    • the need to respect any confidential or sensitive information shared during the conversation.

Meet Your Client (Continued)

Post-Interaction Reflection:

Reflective Practice:

  • Review and Reflect:
    • review their notes and reflect on the conversation soon after,
    • identifying key learnings and any follow-up actions they might take.
  • Feedback Loop:
    • consider what went well and what could be improved for future interactions
    • mindset of continuous improvement.

Mock Interview (If we have time ~ 15min)

  • one-on-one practice: professional vs. student interviewer
  • work together and determine who the professional will shadow (a real person)
  • KYC on industry/persona individually
  • student interviewer to come up with 3 questions as structured interview and one follow-up during the conversation
  • student/professional to swap (if we have time)

Seminar Time: Back to Report Submission with Doc 0.1

Activities

  1. Vote to review one literature review that was submitted (anonymized on mentimeter).
  2. Selected literature reviewed by peers with the following criteria:
  • Objective: Does the literature review sufficiently present its objective?
  • Landscape: Do you think the literature review presented a clear theoretical framework of what are the relevant studies and state-of-the art research landscape?
  • Caveat/Importance: Does the literature review present any clear indication that there is room for investigation of any existing caveats?
  • Citations: Do you think the number of citations included is enough/adequate?
  • Improvements: Anything that the author can do to strengthen the literature review?
  • Challenge: On the topics that were discussed in the literature review, can you quickly leverage google scholar to find additional ones that investigates similar things?
  1. We will review the remainder three of the five documents (so long as time permits) for this week.

Addendum: Checklist for successful literature review

1. Clarity and Coherence:

  • Clear Objective: Define the specific goals and scope of the literature review.
  • Logical Structure: Organize content in a logical manner, facilitating easy navigation and understanding.
  • Language and Terminology: Use clear, concise language appropriate for an interdisciplinary audience, avoiding unnecessary jargon.

2. Problem Statement and Importance:

  • Well-defined Issue: Clearly articulate the problem or research question the literature review addresses.
  • Significance: Explain the importance and relevance of the problem within the context of design and computer science.
  • Gap Identification: Highlight gaps or shortcomings in existing research that the literature review aims to address.

Addendum: Checklist for successful literature review (Cont’d)

3. Theoretical Framework:

  • Conceptual Underpinnings: Present the theories or models that underlie the research area.
  • Framework Integration: Demonstrate how the theoretical framework informs the literature review’s approach and analysis.
  • Interdisciplinary Relevance: Ensure the framework is relevant and accessible to both design and computer science perspectives.

4. Comprehensive Coverage:

  • Breadth and Depth: Cover a wide range of sources while diving deep into critical studies.
  • Diverse Sources: Include academic journals, conference papers, books, and reputable online resources relevant to both fields.
  • Timeliness: Ensure the inclusion of both foundational texts and recent research to reflect the current state of knowledge.

5. Critical Analysis:

  • Comparative Analysis: Compare and contrast different studies, highlighting similarities and differences.
  • Methodological Evaluation: Assess the methodologies used in key studies for their strengths and limitations.
  • Theoretical Critique: Critically evaluate the theories discussed in the literature for their applicability and limitations in the interdisciplinary context.

Addendum: Checklist for successful literature review (Cont’d)

6. Synthesis:

  • Thematic Organization: Synthesize literature thematically rather than summarizing each source individually.
  • Insight Generation: Derive new insights or perspectives from the synthesis of the reviewed literature.
  • Interdisciplinary Integration: Fuse insights from design and computer science to create a cohesive understanding.

7. Relevance to Research Question:

  • Alignment: Ensure all reviewed literature contributes to answering the research question or addressing the problem statement.
  • Application: Discuss how findings from the literature review apply to the specific intersection of design and computer science.

8. Source Evaluation:

  • Credibility Assessment: Evaluate the credibility and reliability of each source.
  • Bias and Perspective: Acknowledge potential biases in the literature and strive for a balanced perspective.

Addendum: Checklist for successful literature review (Cont’d)

9. Conclusions and Implications:

  • Summary of Findings: Concisely summarize key findings and their implications for the research area.
  • Future Research Directions: Identify areas where further research is needed, especially at the intersection of design and computer science.

10. Documentation and Referencing:

  • Consistent Formatting: Adhere to a consistent citation style appropriate for the interdisciplinary audience.
  • Accurate Citations: Ensure all sources are accurately cited within the text and in the reference list.

11. Reflection on Interdisciplinarity:

  • Integration Challenges: Discuss any challenges encountered in integrating design and computer science literature and how they were addressed.
  • Value of Interdisciplinary Approach: Reflect on how the interdisciplinary approach enriches the understanding of the topic.

Revisiting Report Submission

  • Document 0: Research Statement (Extended Abstract)
  • Document 0.1: Literature Review
  • Document 0.2: Methodology & Data Needed/Collected
  • Document 1: Research Proposal (Paper-like)

Check Moodle for current due dates.

Research Process in Flow Chart

Checklist for your final research proposal/paper submission

Elements that you’ve finalized:

  • Research Problem Definition (Doc 0):
    • Articulated innovative challenge or aspect addressed.
    • Contextualized within industry/societal needs.
  • Relevant Concepts and Theories (Doc 0.1):
    • Reviewed key theories underpinning the innovation area.
    • Included interdisciplinary approaches.
  • Previous Research Findings (Doc 0.1):
    • Highlighted past innovations and research.
    • Noted successes and gaps for building upon/addressing.

Ready to be finalized:

  • Hypothesis Formulation (Doc 0.2):
    • Developed clear, testable hypotheses predicting innovation outcomes.
  • Research Design Formulation (Doc 0.2):
    • Outlined appropriate research methodologies/technologies to be used.
    • Discussed feasibility and potential impact of proposed research design.

Research Progress Check

Insofar, Your Design Should Provide:

  • Evidence for Hypothesis:
    • Detailed data collection/analysis plan for hypothesis testing.
    • Consideration on data sources/appropriate technologies deployed in their collection.
  • Significance:
    • Connected significance to societal/industry/academic trends.
    • Highlighted contribution to knowledge/innovation advancement.
  • Implementation and Scalability:
    • Discuss real-world implementation pathways.
    • Address potential scalability barriers.
  • Ethical Considerations and Sustainability:
    • Outline ethical considerations of the innovation.
    • Address sustainability of the proposed solution.

Final Step in Finalizing Research

As Finalizing the Proposal (Doc 1):

  • Integration and Synthesis:
    • Ensured alignment of all proposal components with innovation/research goals.
  • Stakeholder Engagement:
    • Identified key stakeholders: beneficiaries, participants, impacted communities.
    • Included stakeholder engagement plans: evidencing the effectiveness of proposed solution/design
  • Consideration of Future Directions:
    • Speculated on future research directions to further the innovation.

From Data to Insights

The Analysis Gap

You’ve collected data. Now what?

  • Interviews completed → pile of transcripts
  • Survey closed → spreadsheet of responses
  • Observations done → pages of field notes

The hard part isn’t collecting data. It’s making sense of it.

This section bridges that gap.

Thematic Analysis: The 5-Step Process

The most common approach for qualitative data:

Step What You Do Output
1. Familiarize Read everything twice. Note first impressions. Margin notes
2. Code Label meaningful chunks (phrases, sentences) Code list
3. Search for themes Group related codes together Theme candidates
4. Review themes Check if themes hold up across data Refined themes
5. Define & name Write 1-sentence definition for each theme Final themes

Step 1: Familiarize — Read Without Agenda

First pass: Read everything without highlighting. Just absorb.

Second pass: Note anything that: - Surprises you - Repeats across participants - Contradicts what you expected - Feels emotionally charged

Anti-pattern: Jumping straight to coding. You’ll miss the forest for the trees.

Step 2: Code — Label Meaningful Chunks

A code = short label for a piece of data

Quote: > “I check my likes within 10 minutes of posting. I literally can’t help it. It’s like my thumb just goes there.”

Possible codes: - Compulsive checking - Loss of control - Automatic behavior

Code types:

  • Descriptive: What’s happening
  • In-vivo: Participant’s own words
  • Process: Actions/verbs
  • Emotion: Feelings expressed

Step 3: Search for Themes — Cluster Codes

Spread your codes out (physically or digitally). Look for patterns.

Code Cluster Potential Theme
Compulsive checking, can’t stop, automatic Checking as habit loop
Feel bad, anxious, disappointed Emotional cost of low engagement
Compare to friends, feel behind, jealous Social comparison trap
Delete posts, edit captions, time posts Performance management strategies

A theme is not a code. A theme is a pattern that says something meaningful about your research question.

Step 4: Review Themes — Stress Test

For each candidate theme, ask:

Kill themes that don’t hold up. 3-5 strong themes > 10 weak ones.

Step 5: Define & Name — Make It Stick

Each final theme needs:

  1. A clear name (not jargon, not vague)
  2. A 1-sentence definition
  3. Representative quotes (2-3 per theme)
Theme Definition
Compulsive checking loop Participants describe checking likes as automatic, habitual behavior they feel unable to control.
Like thresholds as self-worth Participants set mental benchmarks for “acceptable” like counts; falling short triggers negative self-evaluation.
Performance management Participants actively curate posting behavior (timing, deletion, editing) to optimize engagement metrics.

Worked Example: From 12 Interviews to 4 Themes

Research question: How do HKU students experience Instagram like-checking?

Raw data: 12 interviews, 45 pages of transcripts

Coding: 47 initial codes identified

Final themes:

  1. Compulsive checking loop (10/12 participants)
  2. Like thresholds as self-worth (8/12 participants)
  3. Social comparison with peers (9/12 participants)
  4. Temporary validation effect (7/12 participants)

Each theme has 2-3 supporting quotes and a clear definition.

Architecture/Urban Example: Analyzing Interviews About Public Space

Research question: How do Hong Kong residents perceive rule enforcement in privately-owned public spaces?

Raw data: 8 interviews with regular users of POPS in Central

Sample codes: - “Security told me to move” → enforcement encounter - “I didn’t know I couldn’t sit there” → unclear rules - “It doesn’t feel like public space” → ownership ambiguity - “I just avoid those places now” → behavioral adaptation

Emerging themes:

Theme Definition Prevalence
Invisible boundaries Users discover rules only through enforcement, not signage 6/8
Chilling effect Negative encounters lead to avoidance of space entirely 5/8
Public-private confusion Users unsure who controls space and what rights they have 7/8

So what? Designers should consider how rule communication affects perceived publicness.

For Quantitative Data: From Numbers to Patterns

What You Have What You Do What It Tells You
Survey responses (Likert) Calculate means, SD Central tendency & spread
Two groups to compare t-test or Mann-Whitney Whether difference is real
Two variables Correlation (Pearson/Spearman) Whether they move together
Multiple predictors Regression Which factors matter most

Key question: Is the pattern real, or just noise?

  • p < 0.05 → statistically significant (probably not random)
  • Effect size → practically significant (big enough to matter)

Triangulation: When Qual and Quant Meet

Finding Qual Evidence Quant Evidence Agreement?
Checking frequency → anxiety “I feel anxious when I check” (10/12) r = 0.42, p < .001 ✓ Converge
Likes = self-worth “I feel like a failure” (8/12) Not directly measured Partial
Hiding likes would help “I’d feel relieved” (6/12) 45% selected “would reduce stress” ✓ Converge

When they disagree: That’s interesting data, not a problem. Ask why.

The “So What?” Test

For every finding, ask: Why should anyone care?

Finding So What?
r = 0.42 correlation between checking and anxiety Moderate effect — not everyone, but meaningful signal for intervention design
10/12 describe compulsive checking Design friction (delays, nudges) might break the automatic loop
Social comparison drives checking Hiding like counts removes the comparison trigger

If you can’t answer “so what?” — the finding isn’t ready.

Artifact Output: What You Leave With

Today’s Deliverables

From Part 1 (KYC/Informational Interviews):

  • Professional research profile for 1 potential interviewee
  • 2 informed questions ready to ask

From Part 2 (Analysis):

  • Understanding of the 5-step thematic analysis process
  • Ability to distinguish codes from themes

Take-home thinking: Start coding any interview/observation data you’ve collected. Bring your draft codebook to Week 8.

Learning Objectives

By the end of this session, you will be able to:

  1. Distinguish descriptive from analytical writing in your own drafts
  2. Apply reverse outlining to identify structural gaps
  3. Synthesize qualitative and quantitative findings using the Synthesis Canvas
  4. Translate research findings into product hypotheses (Product Track)

Today’s Session Structure

  1. Writing Workshop (40 min) — Descriptive vs. analytical writing + peer review
  2. Synthesis Activity (25 min) — Working with the Synthesis Canvas
  3. Research → Decision (20 min) — Product Track framework
  4. Logistics (5 min) — Interview project reminder

Descriptive vs. Analytical Writing

Welcome to today’s session on distinguishing between descriptive and analytical writing. Let’s enhance our academic writing skills together!

Activity: Identify the Writing Style

We’ll go through several examples. For each, determine if it’s descriptive or analytical. After your response, we’ll reveal the correct answer and discuss.

Example 1: Question

“User engagement metrics have decreased by 20% over the past three months.”

Question:
Is this statement descriptive or analytical?

Example 1: Answer

Answer: Descriptive

Explanation:
The sentence reports a fact (a 20% drop) without delving into any reasons or implications behind the metric change.

Example 2: Question

“Maslow’s hierarchy of needs is a psychological theory that arranges human needs in a pyramid, with physiological needs at the base and self-actualization at the top.”

Question:
Is this statement descriptive or analytical?

Example 2: Answer

Answer: Descriptive

Explanation:
While it explains the pyramid structure, it does not analyze or critique the theory’s applicability or limitations across different cultures.

Example 3: Question

“The experiment demonstrated that plants exposed to sunlight grew faster than those kept in the shade.”

Question:
Is this statement descriptive or analytical?

Example 3: Answer

Answer: Descriptive

Explanation:
The statement provides an observation (faster growth with sunlight) without discussing the process or significance of the finding.

Example 4: Question

“In Shakespeare’s ‘Macbeth,’ Lady Macbeth’s manipulation of her husband’s actions serves as a commentary on the corrupting power of unchecked ambition and challenges traditional gender roles in Shakespearean society.”

Question:
Is this statement descriptive or analytical?

Example 4: Answer

Answer: Analytical

Explanation:
It not only identifies Lady Macbeth’s influence but also interprets her actions as a critique of societal norms, thus offering analytical insight.

Example 5: Question

“Survey results show that 60% of respondents prefer online shopping over in-store shopping.”

Question:
Is this statement descriptive or analytical?

Example 5: Answer

Answer: Descriptive

Explanation:
It presents a statistic without discussing why users might prefer online shopping or its implications for consumer behavior.

Example 6: Question

“While the interface layout includes a navigation bar, a content area, and a footer, this design might impede user engagement if it fails to prioritize the most critical user tasks, suggesting a need for a more dynamic, behavior-driven layout.”

Question:
Is this statement descriptive or analytical?

Example 6: Answer

Answer: Analytical

Explanation:
The statement goes beyond description by assessing the potential drawbacks of the layout and suggesting improvements based on user behavior.

Key Takeaways

  • Descriptive vs. Analytical:
    • Descriptive writing states facts or observations.
    • Analytical writing explains the significance, reasoning, and implications behind those facts.
  • Depth of Analysis:
    • Effective academic writing connects evidence with critical interpretation.
    • Always ask: “Why is this important?” and “What does this mean for the overall argument?”
  • Practical Strategies for Improvement:
    • Peer Review: Leverage feedback from classmates to identify areas lacking analysis.
    • Reverse Outlining: Break down your text to pinpoint descriptive versus analytical sections.
  • Continuous Revision:
    • Use these insights to refine your drafts, deepening your analysis and improving clarity.

Enhancing Your Analytical Writing

Objective:
- Develop your analytical writing skills by examining and revising your literature review or methodology sections.
- Engage in collaborative activities to practice critical analysis.

Today’s Activities:
1. Peer Review: In pairs, analyze each other’s writing using guided questions.
2. Reverse Outlining: Break down a text (yours or a provided sample) to identify strengths and gaps.

Stage 1: Peer Review Activity

Time: 25 minutes

Instructions:

  1. Pair Up: Find a partner in the room.
  2. Exchange Materials: Share your literature review or methodology section (or use the provided sample).
  3. Guided Review: Use these questions as you read your partner’s work:
    • Thesis & Argument:
      • Is the thesis clear and arguable?
      • Does the argument show critical evaluation rather than just stating facts?
    • Use of Evidence:
      • Is evidence well integrated and analyzed?
      • Does the writer explain the significance of each piece of evidence?
    • Structure & Flow:
      • Are the ideas presented logically?
      • Do transitions connect the analysis throughout the paper?

Goal:
- Provide constructive feedback focused on deepening the analysis.

Stage 2: Reverse Outlining Exercise

Time: 25 minutes

Instructions:

  1. Choose Your Text: Pick a section from the previously worked-upon example you looked at (4-5 paragraphs max).
  2. Number Each Paragraph: Number the paragraphs in the margin for clarity.
  3. Summarize Each Paragraph: In one sentence, capture the main idea or argument of each paragraph.
  4. Evaluate the Outline:
    • Does each summary contribute to your overall argument?
    • Identify any parts that are merely descriptive rather than analytical.
  5. Plan Revisions: Mark paragraphs that need more critical insight or better integration of evidence.

Goal:
- Reveal the structure and flow of your argument and present to each other. - Identify areas where you can add deeper analysis or clarity.

Reflection & Discussion

Discussion Questions:
- What new insights did you gain about your own writing from these activities?
- How can the feedback from your peer review inform your revisions?
- What specific changes will you make to improve the analytical depth of your paper?

Next Steps:
- Revise your draft based on the insights and feedback received. - Schedule a follow-up session if you need additional support on integrating analytical writing techniques.

Double-check your submission to make sure that it can be considered a followable instruction manual by the domain experts.

Synthesis Activity

Using the Synthesis Canvas (25 min)

The challenge: You have findings from multiple sources. How do you bring them together?

  • Qualitative themes from interviews
  • Quantitative patterns from surveys
  • Insights from literature

The tool: Synthesis Canvas (PDF)

Activity: Complete Your Synthesis Canvas

Step 1: What Data Do You Have? (3 min)

Fill in Section 1 of the canvas: - What qualitative data? (interviews, observations, open-ended responses) - What quantitative data? (survey scores, analytics, measurements) - How much of each?

Activity: Qualitative Synthesis (10 min)

Step 2: Pull Key Quotes

From your interviews/observations, identify 3-5 powerful quotes that capture key insights.

Step 3: Group Into Themes

Cluster related quotes. Name each theme. Note how many participants expressed it.

Theme Codes Included # of Participants
Theme 1: _________ _________ /
Theme 2: _________ _________ /

Activity: Triangulation Check (7 min)

Step 4: Do Qual and Quant Agree?

For each finding, check:

Finding Qual Evidence Quant Evidence Agreement?
_________ _________ _________ Yes / No / Partial

If they disagree: That’s interesting data. Write down why they might differ.

Activity: The “So What?” (5 min)

Step 5: Why Should Anyone Care?

For your top 2-3 findings, write one sentence explaining the implication:

Finding So What?
_________ This means… / This suggests… / Designers should consider…

If you can’t answer “so what?” — the finding isn’t ready for Doc 1.

Architecture/Urban Example: Synthesizing Mixed Methods Data

Research question: Does greenery in Hong Kong’s elevated walkways affect pedestrian stress levels?

Data Type Source Key Finding
Qualitative 10 walking interviews “The plants make it feel less like a tunnel” (7/10)
Quantitative Heart rate variability (N=45) 12% lower stress markers in green sections vs. bare sections
Observation Pedestrian counts 23% more people paused/lingered in green sections

Triangulation: All three methods converge — greenery correlates with lower stress and changed behavior.

So what? Urban designers should prioritize vegetation in elevated walkway design; the effect is measurable, not just aesthetic preference.

Research → Decision Framework

Product Track Focus

This section is specifically for students on the Product Track who are building design case studies rather than academic papers. If you’re on the Research Track, this framework shows how industry applies research — useful context even if your output is a traditional paper.

The Product Builder’s Challenge

You’ve done your research. You found the behavioral science. Now what?

Academic output: “Our findings suggest that social comparison mediates the relationship between like-checking frequency and anxiety (β = 0.38, p < .001).”

Product question: “Should we hide like counts? Will it actually help? How will we know?”

This section bridges that gap.

The Translation Problem

What Research Gives You What Product Needs
Correlations & themes Causal hypotheses
General principles Specific feature specs
“This is happening” “If we do X, Y will happen”
Academic language Stakeholder-ready framing

Your job: Turn “what we know” into “what we should build and why we expect it to work.”

The Research → Decision Canvas

Component Question Your Answer
Scientific Foundation What does behavioral science say about this problem? _____________
Product Hypothesis If we build [feature], then [outcome] because [mechanism] _____________
Expected Effect How big a change do we expect? What’s the baseline? _____________
Success Metric How will we measure if it worked? _____________
Risk if Wrong What happens if our hypothesis is wrong? _____________
Validation Plan How will we test before full launch? _____________

Worked Example: Instagram Hidden Likes

Research finding: Social comparison theory (Festinger, 1954) + your qualitative data showing 9/12 participants check likes to compare with friends.

Component Answer
Scientific Foundation Social comparison drives anxiety; visible metrics enable comparison; removing visibility should reduce comparison triggers
Product Hypothesis If we hide like counts from viewers, then comparison-driven anxiety will decrease, because users can’t compare their posts to others’
Expected Effect Baseline: 67% report checking to compare. Target: <40% report comparison motivation after hiding.
Success Metric Self-reported comparison frequency (survey); anxiety scale (GAD-7); qualitative interviews
Risk if Wrong Users might feel less validated, reducing posting motivation; engagement metrics might drop
Validation Plan A/B test with 10% of users for 4 weeks; exit survey + 10 interviews

Case Study: Duolingo’s Streak Feature

The science: Loss aversion (Kahneman & Tversky, 1979) — people are more motivated to avoid losing something than to gain something of equal value.

The translation:

Component Duolingo’s Answer
Scientific Foundation Loss aversion: losing a streak feels worse than gaining a day
Product Hypothesis If we visualize consecutive-day streaks, users will practice daily to avoid “losing” their streak
Expected Effect Daily active users should increase; session frequency should stabilize
Success Metric DAU/MAU ratio; 7-day retention; streak length distribution
Risk if Wrong Users might feel punished by broken streaks and churn
Validation Plan Introduced incrementally; added “streak freeze” as safety valve

Result: Streaks became Duolingo’s core retention mechanism.

Case Study: Spotify Wrapped

The science: Self-presentation theory (Goffman, 1959) — people curate identity through what they share; nostalgia increases emotional engagement.

The translation:

Component Spotify’s Answer
Scientific Foundation Sharing music = identity signaling; year-end reflection = nostalgia; social proof drives adoption
Product Hypothesis If we create shareable, personalized year-in-review content, users will spread it organically, driving brand awareness and emotional connection
Expected Effect Viral social sharing in December; increased premium conversions post-Wrapped
Success Metric Social shares; hashtag volume; December premium signups vs. baseline
Risk if Wrong Users might find it creepy or ignore it
Validation Plan Started with simple stats; iterated based on what got shared most

Result: Wrapped became Spotify’s biggest annual marketing moment — built on behavioral science, not ad spend.

When Your Research Says “Don’t Build It”

Sometimes research reveals your hypothesis is wrong. That’s valuable.

Signals to reconsider:

  • Literature shows the mechanism doesn’t work the way you thought
  • Qualitative data shows users don’t actually have the problem you assumed
  • Comparable products tried it and failed (with documented reasons)
  • The effect size in existing research is too small to matter

What to do:

  1. Document why you’re not building it (prevents revisiting the same dead end)
  2. Look for adjacent opportunities the research revealed
  3. Share with stakeholders — “We saved X months by researching first”

What Product Research Output Looks Like

Not a paper. A decision document.

Structure:

  1. The Opportunity (1 paragraph): What problem are we solving? For whom?
  2. What We Learned (1 page): Key findings from research — qual themes, quant patterns, literature insights
  3. Our Hypothesis (1 paragraph): If we build X, then Y, because Z
  4. Recommendation (1 paragraph): Build / Don’t build / Test first
  5. Validation Plan (if building): How we’ll know if we’re right
  6. Appendix: Raw data, quotes, methodology details

Length: 2-3 pages + appendix. Readable in 10 minutes.

The Screenshot-Worthy Slide

Research → Decision in One Sentence

“We found [scientific principle] in the literature and [pattern] in our data, which suggests that if we build [feature], users will [outcome] — and we’ll measure success by [metric].”

Example:

“We found that loss aversion (Kahneman & Tversky) predicts people work harder to protect what they have than to gain something new. Our interviews confirmed users hate ‘losing progress.’ This suggests that if we add a visible streak counter, users will practice daily to avoid breaking it — and we’ll measure success by 7-day retention rate.”

Artifact Output: What You Leave With

Today’s Deliverables

From Writing Workshop:

  • Reverse outline of your Doc 0.1 or Doc 0.2 draft
  • Peer feedback on descriptive vs. analytical writing
  • List of paragraphs that need deeper analysis

From Synthesis (Research Track):

  • Draft Synthesis Canvas with at least 2 themes + triangulation check

From Research → Decision (Product Track):

  • Completed Research to Decision Canvas with hypothesis and validation plan

These feed directly into Doc 1.

Logistics: Interview Project Reminder

Group/Individual Interview Project

Quick Reminder

The interview-based project (group or individual) gives you practice with:

  • Professional outreach and networking
  • Primary data collection through interviews
  • Synthesizing insights from first-hand sources

Choose group or individual based on your preferences. Check Moodle for current deadlines and submission guidelines.

A foundational course in research methodology for design practice. We explore how rigorous inquiry informs innovation—from framing questions to gathering evidence and communicating findings.

The Course’s Core Question

Can you distinguish incremental improvement from genuine innovation? Research methodology gives you the tools to tell the difference — and to pursue whichever path you choose deliberately.

Two Tracks, One Methodology:

Research Track Product Track
Extended Abstract → Literature Review → Methodology → Research Paper Product Research Brief → Landscape Intelligence → Research Playbook → Design Case Study

Choose based on your interests. Same rigor. Same skills. Different outputs.

Key Resources

Templates & Tools

Printable PDF toolkits — each includes a blank template on page 1 and a worked example on page 2.

Week Toolkit Description
3 Landscape Intelligence Canvas Competitive + academic research synthesis
4 Research Question Builder FINER criteria, scope check, question types
5 Interview Protocol Template Opening script, probes, anti-patterns
6 Survey Design Checklist Bias checks, Likert scales, sample size guide
9 Synthesis Canvas Qual→themes, quant patterns, triangulation
9 Research to Decision Canvas For product builders: research → feature hypothesis

Other Resources:

Lectures & Materials

Course Info

Schedule Tuesday, 1:00–2:50 PM

Location KKLG101

Office hours by appointment.